Identifying Psychosis Episodes in Psychiatric Admission Notes via Rule-based Methods, Machine Learning, and Pre-Trained Language Models.

Early and accurate diagnosis is crucial for effective treatment and improved outcomes, yet identifying psychotic episodes presents significant challenges due to its complex nature and the varied presentation of symptoms among individuals. One of the primary difficulties lies in the underreporting and underdiagnosis of psychosis, compounded by the stigma surrounding mental health and the individuals' often diminished insight into their condition. Existing efforts leveraging Electronic Health Records (EHRs) to retrospectively identify psychosis typically rely on structured data, such as medical codes and patient demographics, which frequently lack essential information. Addressing these challenges, our study leverages Natural Language Processing (NLP) algorithms to analyze psychiatric admission notes for the diagnosis of psychosis, providing a detailed evaluation of rule-based algorithms, machine learning models, and pre-trained language models. Additionally, the study investigates the effectiveness of employing keywords to streamline extensive note data before training and evaluating the models. Analyzing 4,617 initial psychiatric admission notes (1,196 cases of psychosis versus 3,433 controls) from 2005 to 2019, we discovered that the XGBoost classifier employing Term Frequency-Inverse Document Frequency (TF-IDF) features derived from notes pre-selected by expert-curated keywords, attained the highest performance with an F1 score of 0.8881 (AUROC [95% CI]: 0.9725 [0.9717, 0.9733]). BlueBERT demonstrated comparable efficacy an F1 score of 0.8841 (AUROC [95% CI]: 0.97 [0.9580, 0.9820]) on the same set of notes. Both models markedly outperformed traditional International Classification of Diseases (ICD) code-based detection methods from discharge summaries, which had an F1 score of 0.7608, thus improving the margin by 0.12. Furthermore, our findings indicate that keyword pre-selection markedly enhances the performance of both machine learning and pre-trained language models. This study illustrates the potential of NLP techniques to improve psychosis detection within admission notes and aims to serve as a foundational reference for future research on applying NLP for psychosis identification in EHR notes.

Dynamic reaction picklist for improving allergy reaction documentation: A usability study.

OBJECTIVE: To assess novel dynamic reaction picklists for improving allergy reaction documentation compared to a static reaction picklist. MATERIALS AND METHODS: We developed three web-based user interfaces (UIs) mimicking the Mass General Brigham's EHR allergy module: the first and second UIs (i.e., UI-1D, UI-2D) implemented two dynamic reaction picklists with different ranking algorithms and the third UI (UI-3S) implemented a static reaction picklist like the one used in the current EHR. We recruited 18 clinicians to perform allergy entry for 10 test cases each via UI-1D and UI-3S, and another 18 clinicians via UI-2D and UI-3S. Primary measures were the number of free-text entries and time to complete the allergy entry. Clinicians were also interviewed using 30 questions before and after the data entry. RESULTS AND DISCUSSIONS: Among 36 clinicians, less than half were satisfied with the current EHR reaction picklists, due to their incomprehensiveness, inefficiency, and lack of intuitiveness. The clinicians used significantly fewer free-text entries when using UI-1D or UI-2D compared to UI-3S (p < 0.05). The clinicians used on average 51 s (15 %) less time via UI-1D and 50 s (16 %) less time via UI-2D in completing the allergy entries versus UI-3S, and there was not a statistically significant difference in documentation time for either group between the dynamic and static UIs. Overall, 15-17 (83-94 %) clinicians rated UI-1D and 13-15 (72-83 %) clinicians rated UI-2D as efficient, easy to use, and useful, while less than half rated the same for UI-3S. Most clinicians reported that the dynamic reaction picklists always or often suggested appropriate reactions (n = 30, 83 %) and would decrease the free-text entries (n = 26, 72 %); nearly all preferred the dynamic picklist over the static picklist (n = 32, 89 %). CONCLUSION: We found that dynamic reaction picklists significantly reduced the number of free-text entries and could reduce the time for allergy documentation by 15%. Clinicians preferred the dynamic reaction picklist over the static picklist.

Identifying and Reconciling Patients' Allergy Information Within the Electronic Health Record.

Allergy information is often documented in diverse sections of the electronic health record (EHR). Systematically reconciling allergy information across the EHR is critical to improve the accuracy and completeness of patients' allergy lists and ensure patient safety. In this retrospective cohort study, we examined the prevalence of incompleteness, inaccuracy, and redundancy of allergy information for patients with a clinical encounter at any Mass General Brigham facility between January 1, 2018 and December 31, 2018. We identified 4 key places in the EHR containing reconcilable allergy information: 1) allergy modules (including free text comments and duplicate allergen entries), 2) medication laboratory tests results, 3) oral medication allergy challenge tests, and 4) medication orders that have been discontinued due to adverse drug reactions (ADRs). Within our cohort, 718,315 (45.2% of the total 1,588,979) patients had an active allergy entry; of which, 266,275 (37.1%) patient's records indicated a need for reconciliation.

Heterogeneity of Drug Allergies and Reaction Lists in Two U.S. Health Care Systems' Electronic Health Records.

BACKGROUND: Health care institutions have their own "picklist" for clinicians to document adverse drug reactions (ADRs) into the electronic health record (EHR) allergy list. Whether the lack of a nationally standardized picklist impacts clinician data entries is unknown. OBJECTIVES: The objective of this study was to assess the impact of defined reaction picklists on clinical documentation and, therefore, downstream analytics and clinical research using these data at two institutions. METHODS: ADR data were obtained from the EHRs of patients who visited the emergency department or outpatient clinics at Brigham and Women's Hospital (BWH) and University of Colorado Hospital (UCH) from 2013 to 2018. Reported drug class ADR prevalences were calculated. We investigated the reactions on each picklist and compared the top 40 reactions at each institution, as well as the top 10 reactions within each drug class. RESULTS: Of 2,160,116 patients, 640,444 (30%) had 928,973 active drug allergies. The most commonly reported drug class allergens were similar between BWH and UCH. BWH's picklist had 48 reactions, and UCH's had 160 reactions; 29 reactions were shared by both picklists. While the top four reactions overall (rash, GI upset/nausea/vomiting, hives, itching) were identical between sites, reactions by drug class exhibited greater documentation diversity. For example, while the summed prevalence of swelling-related reactions to angiotensin-converting-enzyme inhibitors was comparable across sites, swelling was represented by two terms ("swelling," "angioedema") at BWH but 11 terms at UCH (e.g., "swelling," "edema," by body locality). CONCLUSION: The availability and granularity of reaction picklists impact ADR documentation in the EHR by health care providers; picklists may partially explain variations in reported ADRs across health care systems.

Vancomycin Hypersensitivity Reactions Documented in Electronic Health Records.

BACKGROUND: Vancomycin, the most common antimicrobial used in US hospitals, can cause diverse adverse reactions, including hypersensitivity reactions (HSRs). Yet, little is known about vancomycin reactions documented in electronic health records. OBJECTIVE: To describe vancomycin HSR epidemiology from electronic health record allergy data. METHODS: This was a cross-sectional study of patients with 1 or more encounter from 2017 to 2019 and an electronic health record vancomycin drug allergy label (DAL) in 2 US health care systems. We determined prevalence and trends of vancomycin DALs and assessed active DALs by HSR phenotype determined from structured (coded) and unstructured (free-text) data using natural language processing. We investigated demographic associations with documentation of vancomycin red man syndrome (RMS). RESULTS: Among 4,490,618 patients, 14,426 (0.3%) had a vancomycin DAL with 18,761 documented reactions (2,248 [12.0%] free-text). Quarterly mean vancomycin DALs added were 253 ± 12 and deleted were 12 ± 2. Of 18,761 vancomycin HSRs, 7,903 (42.1%) were immediate phenotypes and 3,881 (20.7%) were delayed phenotypes. Common HSRs were rash (32% of HSRs) and RMS (16% of HSRs). Anaphylaxis was coded in 6% cases of HSRs. Drug reaction eosinophilia and systemic symptoms syndrome was the most common coded vancomycin severe cutaneous adverse reaction. RMS documentation was more likely for males (odds ratio, 1.30; 95% CI, 1.17-1.44) and less likely for blacks (odds ratio, 0.59; 95% CI, 0.47-0.75). CONCLUSIONS: Vancomycin causes diverse adverse reactions, including common (eg, RMS) and severe (eg, drug reaction eosinophilia and systemic symptoms syndrome) reactions entered as DAL free-text. Anaphylaxis comprised 6% of documented vancomycin HSRs, although true vancomycin IgE-mediated reactions are exceedingly rare. Improving vancomycin DAL documentation requires more coded entry options, including a coded entry for RMS.

An annotated dataset of tongue images supporting geriatric disease diagnosis.

Hospitalized geriatric patients are a highly heterogeneous group often with variable diseases and conditions. Physicians, and geriatricians especially, are devoted to seeking non-invasive testing tools to support a timely, accurate diagnosis. Chinese tongue diagnosis, mainly based on the color and texture of the tongue, offers a unique solution. To develop a non-invasive assessment tool using machine learning in supporting a timely, accurate diagnosis in the elderly, we created an annotated dataset of 15% of 688 (=100) tongue images collected from hospitalized geriatric patients in a tertiary hospital in Shanghai, China. Images were captured via a light-field camera using CIELAB color space (to simulate human visual perception) and then were manually labeled by a panel of subject matter experts after chart reviewing patients' clinical information documented in the hospital's information system. We expect that the dataset can assist in implementing a systematic means of conducting Chinese tongue diagnosis, predicting geriatric syndromes using tongue appearance, and even developing an mHealth application to provide individualized health suggestions for the elderly.

Physician use of speech recognition versus typing in clinical documentation: A controlled observational study.

IMPORTANCE: Speech recognition (SR) is increasingly used directly by clinicians for electronic health record (EHR) documentation. Its usability and effect on quality and efficiency versus other documentation methods remain unclear. OBJECTIVE: To study usability and quality of documentation with SR versus typing. DESIGN: In this controlled observational study, each subject participated in two of five simulated outpatient scenarios. Sessions were recorded with Morae® usability software. Two notes were documented into the EHR per encounter (one dictated, one typed) in randomized order. Participants were interviewed about each method's perceived advantages and disadvantages. Demographics and documentation habits were collected via survey. Data collection occurred between January 8 and February 8, 2019, and data analysis was conducted from February through September of 2019. SETTING: Brigham and Women's Hospital, Boston, Massachusetts, USA. PARTICIPANTS: Ten physicians who had used SR for at least six months. MAIN OUTCOMES AND MEASURES: Documentation time, word count, vocabulary size, number of errors, number of corrections and quality (clarity, completeness, concision, information sufficiency and prioritization). RESULTS: Dictated notes were longer than typed notes (320.6 vs. 180.8 words; p = 0.004) with more unique words (170.9 vs. 120.4; p = 0.01). Documentation time was similar between methods, with dictated notes taking slightly less time to complete than typed notes. Typed notes had more uncorrected errors per note than dictated notes (2.9 vs. 1.5), although most were minor misspellings. Dictated notes had a higher mean quality score (7.7 vs. 6.6; p = 0.04), were more complete and included more sufficient information. CONCLUSIONS AND RELEVANCE: Participants felt that SR saves them time, increases their efficiency and allows them to quickly document more relevant details. Quality analysis supports the perception that SR allows for more detailed notes, but whether dictation is objectively faster than typing remains unclear, and participants described some scenarios where typing is still preferred. Dictation can be effective for creating comprehensive documentation, especially when physicians like and feel comfortable using SR. Research is needed to further improve integration of SR with EHR systems and assess its impact on clinical practice, workflows, provider and patient experience, and costs.

A dynamic reaction picklist for improving allergy reaction documentation in the electronic health record.

OBJECTIVE: Incomplete and static reaction picklists in the allergy module led to free-text and missing entries that inhibit the clinical decision support intended to prevent adverse drug reactions. We developed a novel, data-driven, "dynamic" reaction picklist to improve allergy documentation in the electronic health record (EHR). MATERIALS AND METHODS: We split 3 decades of allergy entries in the EHR of a large Massachusetts healthcare system into development and validation datasets. We consolidated duplicate allergens and those with the same ingredients or allergen groups. We created a reaction value set via expert review of a previously developed value set and then applied natural language processing to reconcile reactions from structured and free-text entries. Three association rule-mining measures were used to develop a comprehensive reaction picklist dynamically ranked by allergen. The dynamic picklist was assessed using recall at top k suggested reactions, comparing performance to the static picklist. RESULTS: The modified reaction value set contained 490 reaction concepts. Among 4 234 327 allergy entries collected, 7463 unique consolidated allergens and 469 unique reactions were identified. Of the 3 dynamic reaction picklists developed, the 1 with the optimal ranking achieved recalls of 0.632, 0.763, and 0.822 at the top 5, 10, and 15, respectively, significantly outperforming the static reaction picklist ranked by reaction frequency. CONCLUSION: The dynamic reaction picklist developed using EHR data and a statistical measure was superior to the static picklist and suggested proper reactions for allergy documentation. Further studies might evaluate the usability and impact on allergy documentation in the EHR.

Using Natural Language Processing and Machine Learning to Identify Hospitalized Patients with Opioid Use Disorder.

Opioid use disorder (OUD) represents a global public health crisis that challenges classic clinical decision making. As existing hospital screening methods are resource-intensive, patients with OUD are significantly under-detected. An automated and accurate approach is needed to improve OUD identification so that appropriate care can be provided to these patients in a timely fashion. In this study, we used a large-scale clinical database from Mass General Brigham (MGB; formerly Partners HealthCare) to develop an OUD patient identification algorithm, using multiple machine learning methods. Working closely with an addiction psychiatrist, we developed a set of hand-crafted rules for identifying information suggestive of OUD from free-text clinical notes. We implemented a natural language processing (NLP)-based classification algorithm within the Medical Text Extraction, Reasoning and Mapping System (MTERMS) tool suite to automatically label patients as positive or negative for OUD based on these rules. We further used the NLP output as features to build multiple machine learning and a neural classifier. Our methods yielded robust performance for classifying hospitalized patients as positive or negative for OUD, with the best performing feature set and model combination achieving an F1 score of 0.97. These results show promise for the future development of a real-time tool for quickly and accurately identifying patients with OUD in the hospital setting.

A clinician survey of using speech recognition for clinical documentation in the electronic health record.

OBJECTIVE: To assess the role of speech recognition (SR) technology in clinicians' documentation workflows by examining use of, experience with and opinions about this technology. MATERIALS AND METHODS: We distributed a survey in 2016-2017 to 1731 clinician SR users at two large medical centers in Boston, Massachusetts and Aurora, Colorado. The survey asked about demographic and clinical characteristics, SR use and preferences, perceived accuracy, efficiency, and usability of SR, and overall satisfaction. Associations between outcomes (e.g., satisfaction) and factors (e.g., error prevalence) were measured using ordinal logistic regression. RESULTS: Most respondents (65.3%) had used their SR system for under one year. 75.5% of respondents estimated seeing 10 or fewer errors per dictation, but 19.6% estimated half or more of errors were clinically significant. Although 29.4% of respondents did not include SR among their preferred documentation methods, 78.8% were satisfied with SR, and 77.2% agreed that SR improves efficiency. Satisfaction was associated positively with efficiency and negatively with error prevalence and editing time. Respondents were interested in further training about using SR effectively but expressed concerns regarding software reliability, editing and workflow. DISCUSSION: Compared to other documentation methods (e.g., scribes, templates, typing, traditional dictation), SR has emerged as an effective solution, overcoming limitations inherent in other options and potentially improving efficiency while preserving documentation quality. CONCLUSION: While concerns about SR usability and accuracy persist, clinicians expressed positive opinions about its impact on workflow and efficiency. Faster and better approaches are needed for clinical documentation, and SR is likely to play an important role going forward.

Speech recognition for clinical documentation from 1990 to 2018: a systematic review.

OBJECTIVE: The study sought to review recent literature regarding use of speech recognition (SR) technology for clinical documentation and to understand the impact of SR on document accuracy, provider efficiency, institutional cost, and more. MATERIALS AND METHODS: We searched 10 scientific and medical literature databases to find articles about clinician use of SR for documentation published between January 1, 1990, and October 15, 2018. We annotated included articles with their research topic(s), medical domain(s), and SR system(s) evaluated and analyzed the results. RESULTS: One hundred twenty-two articles were included. Forty-eight (39.3%) involved the radiology department exclusively and 10 (8.2%) involved emergency medicine; 10 (8.2%) mentioned multiple departments. Forty-eight (39.3%) articles studied productivity; 20 (16.4%) studied the effect of SR on documentation time, with mixed findings. Decreased turnaround time was reported in all 19 (15.6%) studies in which it was evaluated. Twenty-nine (23.8%) studies conducted error analyses, though various evaluation metrics were used. Reported percentage of documents with errors ranged from 4.8% to 71%; reported word error rates ranged from 7.4% to 38.7%. Seven (5.7%) studies assessed documentation-associated costs; 5 reported decreases and 2 reported increases. Many studies (44.3%) used products by Nuance Communications. Other vendors included IBM (9.0%) and Philips (6.6%); 7 (5.7%) used self-developed systems. CONCLUSION: Despite widespread use of SR for clinical documentation, research on this topic remains largely heterogeneous, often using different evaluation metrics with mixed findings. Further, that SR-assisted documentation has become increasingly common in clinical settings beyond radiology warrants further investigation of its use and effectiveness in these settings.

Analysis of Errors in Dictated Clinical Documents Assisted by Speech Recognition Software and Professional Transcriptionists.

IMPORTANCE: Accurate clinical documentation is critical to health care quality and safety. Dictation services supported by speech recognition (SR) technology and professional medical transcriptionists are widely used by US clinicians. However, the quality of SR-assisted documentation has not been thoroughly studied. OBJECTIVE: To identify and analyze errors at each stage of the SR-assisted dictation process. DESIGN SETTING AND PARTICIPANTS: This cross-sectional study collected a stratified random sample of 217 notes (83 office notes, 75 discharge summaries, and 59 operative notes) dictated by 144 physicians between January 1 and December 31, 2016, at 2 health care organizations using Dragon Medical 360 | eScription (Nuance). Errors were annotated in the SR engine-generated document (SR), the medical transcriptionist-edited document (MT), and the physician's signed note (SN). Each document was compared with a criterion standard created from the original audio recordings and medical record review. MAIN OUTCOMES AND MEASURES: Error rate; mean errors per document; error frequency by general type (eg, deletion), semantic type (eg, medication), and clinical significance; and variations by physician characteristics, note type, and institution. RESULTS: Among the 217 notes, there were 144 unique dictating physicians: 44 female (30.6%) and 10 unknown sex (6.9%). Mean (SD) physician age was 52 (12.5) years (median [range] age, 54 [28-80] years). Among 121 physicians for whom specialty information was available (84.0%), 35 specialties were represented, including 45 surgeons (37.2%), 30 internists (24.8%), and 46 others (38.0%). The error rate in SR notes was 7.4% (ie, 7.4 errors per 100 words). It decreased to 0.4% after transcriptionist review and 0.3% in SNs. Overall, 96.3% of SR notes, 58.1% of MT notes, and 42.4% of SNs contained errors. Deletions were most common (34.7%), then insertions (27.0%). Among errors at the SR, MT, and SN stages, 15.8%, 26.9%, and 25.9%, respectively, involved clinical information, and 5.7%, 8.9%, and 6.4%, respectively, were clinically significant. Discharge summaries had higher mean SR error rates than other types (8.9% vs 6.6%; difference, 2.3%; 95% CI, 1.0%-3.6%; P < .001). Surgeons' SR notes had lower mean error rates than other physicians' (6.0% vs 8.1%; difference, 2.2%; 95% CI, 0.8%-3.5%; P = .002). One institution had a higher mean SR error rate (7.6% vs 6.6%; difference, 1.0%; 95% CI, -0.2% to 2.8%; P = .10) but lower mean MT and SN error rates (0.3% vs 0.7%; difference, -0.3%; 95% CI, -0.63% to -0.04%; P = .03 and 0.2% vs 0.6%; difference, -0.4%; 95% CI, -0.7% to -0.2%; P = .003). CONCLUSIONS AND RELEVANCE: Seven in 100 words in SR-generated documents contain errors; many errors involve clinical information. That most errors are corrected before notes are signed demonstrates the importance of manual review, quality assurance, and auditing.